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Abstract 

Currently, e-learning systems are mainly web-based applications and tackle a wide range of 
users all over the world. Fitting learners' needs is considered as a key issue to guaranty the 
success of these systems. Many researches work on providing adaptive systems. Nevertheless, 
evaluation of the adaptivity is still in an exploratory phase. Adaptation methods are a basic 
factor to guaranty an effective adaptation. This issue is referred as meta-adaptation in numer¬ 
ous researches. In our research on the development of an evaluation framework of adaptive 
web-based learning systems, adaptation method assessment is a fundamental aspect. Cur¬ 
rently, measures significantly lack to express the adaptive systems features and need to be 
explored. Consequently, we propose a three-fold approach. Firstly, specific adaptation meas¬ 
urement criteria are suggested. Secondly, experts and learners assess these criteria and both 
current learning situation and similar past experiences are considered. Finally, fuzzy group 
decision making theory is adopted to integrate different perceptions related to the adaptive 
system. 
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Introduction 

E-learning is a leader field for adaptive web based systems. The number of stake¬ 
holders is considerably growing all over the world. The IDC (International Data Cor¬ 
poration) forecasts an important increase in the e-learning market: $21 billions in 
2008. These users are different at many levels: Educational goals, social situation, 
language, skills, etc. To cope with their needs, several adaptive Web-based educa¬ 
tional systems are currently available in the market. Nevertheless, many studies 
showed that adaptivity issue is non-sufficiently addressed in these systems (Graf & 
List, 2005; Brusilovsky, 2004; Gutl et al., 2004; Russell, 1999). 

Optimizing adaptivity effectiveness necessitates essentially an objective evaluation 
which indicates the reasons of failures or successes. This idea is the key issue of lay¬ 
ered evaluation approaches (Paramythis etal., 2001; Weibelzahl, 2003; Brusilovsky 
et al., 2004; Paramythis & Weibelzahl, 2005). The latter studies give an overview of 
adaptive system layers to be addressed in the evaluation process. However, specific 
measurement criteria which concretely describe adaptive systems features are sig¬ 
nificantly lacking and there are no evaluation frameworks for measuring the quality 
of adaptation. The main goal of our research is to provide a conceptual framework for 
adaptive learning systems evaluation. Adaptation method is an abstract layer which 
is considered in all adaptive systems. Consequently, it is essential to focus on this is¬ 
sue in the development of adaptive system evaluation framework. 

In this paper, we aim to explore metrics related to the method layer in order to assess 
successes or limitations of methods in each learning context and to be able to choose 
the suitable method in the appropriate situation. The adaptation of adaptive methods 
to a specific context is referred in many researches as meta-adaptation (Brusilovsky, 
2003; Revilla & Shipman, 2003; Hillman & Warren, 2004). 

In our work, both experts and learners judgements are considered. Decisions related 
to the different measures are generally described by qualitative values. Fuzzy group 
decision making theory is adopted to evaluate the global degree of adaptation 
method performance. 

Recent adaptive hypermedia systems are mainly based on AHAM (Adaptive Hyper¬ 
media Application Model) reference model (De Bra et al, 1999). Accordingly, in our 
research, we deal with adaptive systems that follow the principles presented in this 
model. In the following section, AHAM model is briefly introduced. 
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AHAM reference model 

AHAM is a general reference model for adaptive hypermedia systems. It divides the 
adaptive application into three basic components: User Model, Domain Model and 
Adaptation Model. 

Domain Model 

Hypermedia systems are described generally by nodes and links. Adaptive Hyperme¬ 
dia systems, based on AHAM model, deal with concepts and concept relationships. 

Concepts: AHAM introduces three types of concepts: fragments, pages and composite 
concepts. A fragment is an atomic information unit that cannot be modified by the 
adaptive hypermedia system. A page component is a set of fragments that are pre¬ 
sented to the user at the same time. A composite component is a set of pages and/or 
other composite components. The hierarchy of these concepts is presented in Figure 1. 


composite concept 



pages 




F2 F3 F4 F5 

Figure 1. Concepts hierarchy 


fragments 


Concept relationships: Generally, in hypermedia systems, relationship between 
components is based on navigational link only. Nevertheless, in an adaptation con¬ 
text, semantic aspect is considered in concept relationship. Mainly, the following rela¬ 
tionship types are considered in AHAM: 

• Link : present the navigational link between components 

• Prerequisite: designate order of reading between concepts 

• Inhibitor: designate a non-desirable reading order. 

AHAM allows others relationships’ types to be added. 
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Adaptation Model 

Adaptation model describes adaptive hypermedia systems at an abstract conceptual 
level. It uses information from domain model, user model and user interaction to up¬ 
date the user model and to perform adaptation. AHAM adopts language of rules to 
describe the adaptation strategies. The interpreter of these rules is called adaptation 
engine. It uses adaptation rules to generate HTML pages: It builds them by selecting 
and/or sorting fragments. 

User Model 

In order to fit users’ needs, adaptive systems must store information about his or her 
preferences, knowledge, background, environment, etc. AHAM is based essentially on 
user knowledge about the application domain. Consequently, the user model is an 
overlay model of the domain model. For each concept in the domain model, attributes 
values about the concept are stored in the user model, such as: not known, learnt, 
well-known, read, ready to read, etc. 


Adaptation methods evaluation approach 

A system is called adaptive if it is able to change its own characteristics automatically 
according to user’s needs (Oppermann, 1994], Different methods and techniques are 
applied in order to insure adaptivity. The method tackles adaptation in an abstract 
and conceptual level whereas techniques are an implementation of adaptation meth¬ 
ods. Measures used in adaptive systems’ evaluation are mainly borrowed from Hu¬ 
man Computer Interaction [HCI] field and new approaches are strongly called for 
exploring new metrics to assess adaptivity (Gena & Weibelzahl, 2006]. Accordingly, 
we aim to address in this paper these limitations and we focus essentially on adapta¬ 
tion methods in e-learning context. 

Our approach is in three phases. Firstly, we explore measurement criteria associated to 
adaptation method. Secondly, different measures assessments are considered. Fi¬ 
nally, a global evaluation result is generated which integrate the different percep¬ 
tions related to the adaptive system. 

Evaluation criteria exploration 

One adaptation method can not be suitable to all contexts. For this reason, adaptation 
methods should be adapted to the changing context of adaptive learning systems and 
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learners. Current systems don’t deal with meta-adaptation. Results of the proposed 
approach can contribute to integrate meta-adaptation in adaptive systems. 

In AHAM, three basic models can be considered in each adaptive system: adaptation 
model, user model and domain model. Based on this principle, we assume that adap¬ 
tation methods concern, in one hand, user modelling which deduces inferences from 
user interaction with the e-learning system. In the other hand, they are related to 
adaptive content and link generation. Consequently, adaptation methods can be di¬ 
vided into three main categories: user modelling methods, content or presentation 
methods and navigational or link methods. 

The use of a single user model often results in rigid adaptation strategies (Revilla & 
Shipman, 2003). To cope with this difficulty, researchers have augmented this ap¬ 
proach by employing multiple models (Revilla & Shipman, 2000). Different user 
modelling methods can be adopted to create the user model, to analyze the user in¬ 
teraction with adaptive system and to deduce new information about the user. They 
include: 

• Overlay method 

• Stereotype method 

• Bayesian methods 

• Modal logic 

• Machine learning methods, such as features-based techniques, neural networks, 
and explicit user ratings. 

Brusilovsky (2003) has revealed that users with different knowledge level of the sub¬ 
ject may appreciate different adaptive navigation support technologies. The main 
adaptive navigation support methods are the following: 

• Global guidance 

• Local guidance 

• Global orientation support 

• Local orientation support 

• Managing personalized views. 

Presentation concerns the content presented to the adaptive system users. The fol¬ 
lowing adaptive presentation methods can be adopted: 

• Additional, prerequisite, and comparative explanations 
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• Explanation variants 

• Sorting. 

In order to evaluate adaptation methods, we introduce measurement criteria to as¬ 
sess adaptation methods performance. We propose three basic criteria as following: 

Method appropriateness: The choice of the adaptation method affects adaptation 
quality and user satisfaction. Appropriateness criterion deals with which method is 
most appropriate, when and where? Many researches in software engineering tackle 
this problem. The situational approaches bring solutions that may be adapted to 
adaptive systems context. For instance local guidance method in link adaptation 
gives the user an idea of only the following step. But, global guidance gives the user 
an overview of all the following steps. Appropriateness study would help adaptive 
system designers to decide which method is most suitable, when and in which con¬ 
text. 

Exactness of inferences: Adaptation methods update user model based on user in¬ 
teraction with the system. In the other hand, they generate adaptation (content and 
link] deduced from the current state of the user model. Nevertheless, inferences can 
be erroneous, since interpretation of human reaction is complex and not systematic. 
Number of learning concepts can be accessed by user and not learnt or only partially 
learnt. The user can see the web page without learning its concepts or understanding 
them. Although, the adaptation method can declare the concept as learnt and provide 
subsequent adaptation. Exactness of inferences evaluation criterion can be related 
either to user modelling methods or content and navigation methods which make 
inferences based on user model. 

Learner acceptance: Adaptation should be perceived by learner as a natural proc¬ 
ess. Therefore, if the user feels to be spied on or forced to learn a given concept and 
to reach a particular link, then the adaptation method fails to fit the learner needs 
and it is not accepted. Care should be taken that these methods are not too obtrusive 
with respect to the interaction itself (Paramythis et al., 2001], 

Evaluation criteria assessment 

Criteria assessment is performed by learners and experts. In this research, we pro¬ 
pose an adaptation method evaluation based on current measures and past evalua¬ 
tion experiences results. Measurement criteria related to adaptation methods are 
mainly qualitative and subjective. Decisions in this context are often expressed in 
natural language and evaluators are unable to assign exact numerical values to the 
different criteria. Assessment can be possibly performed by linguistic variables like: 
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“bad”, “poor”, “fair”, “good” and “excellent”. These values are imprecise and uncer¬ 
tain: They are commonly called fuzzy values. Integrating these different judgments in 
order to obtain a final evaluation is not evident. We propose to use fuzzy group deci¬ 
sion making theory to obtain final assessment measures related to adaptation meth¬ 
ods. 


Global evaluation based on fuzzy group decision-making theory 


In order to deal with fuzzy multi-criteria and multi-decisions evaluation problem, 
fuzzy group decision-making theory is adopted. 


First, linguistic variable values are mapped into fuzzy numbers. In this paper, we use 
triangular fuzzy numbers (TFNs) which are a class of the fuzzy set representation 
(Dubois &Prade, 1978; Zadeh, 1975a, b, c). A triangular fuzzy number N is expressed 
by three real numbers (l,m,u): The parameters 1, m, and u, respectively, indicate the 
lower, the mean, and the upper possible values. TFNs membership functions are the 
following: 


0 


h(x/ N) = 


J (x-l)/(m-l) 
(u - x) / (u - m] 
0 


x < 1 

1 < x < m 
m < x < u 
x > u 


Figure 2 illustrates the mapping of linguistic variables into the corresponding TFNs. 
In our work we adopt a five criteria assessment scale: "bad”, "poor”, "fair”, "good” and 
"excellent”. Conversion of these qualitative values into fuzzy numbers is showed in 
Table 1. 



Figure 2. The mapping of linguistic variable into the corresponding TFN 
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Table 1. Linguistic variables conversion into TFN 


Linguistic variables 

TFN 

Excellent 

(0.700, 0.850, 1.000) 

Good 

(0.525, 0.675, 0.825) 

Fair 

(0.350, 0.500, 0.650) 

Poor 

(0.175, 0.325, 0.475) 

Bad 

(0.000, 0..150, 0.300) 


In order to compute the fuzzy global evaluation result, first we integrate the different 
evaluators’ fuzzy assessments for each criterion using geometric mean method sug¬ 
gested by (Buckley, 1985): 

Ni = (Nd <8) Ni 2 0....0 Nj m ) V/ m 

Nji: Judgement of the evaluator j towards the criterion i; 

Ni: The average fuzzy number of evaluators’ judgments towards criterion i; 
m: The total number of evaluators; 

0: Fuzzy number multiplication; 

In our study we suppose that all criteria have the same weight. The global fuzzy 
evaluation result is obtained by using arithmetic mean as follow: 

N = 1/n (Ni© N 2 ©...© NO 
n : The total number of criteria; 

© : Fuzzy number addition; 

The last step is the defuzzification procedure. We adopt the center of area (COA) 
method to convert the global fuzzy evaluation result N, expressed by a triangular 
fuzzy number (1, m, u), to a non-fuzzy value E. The following equation is adopted: 

E = [(u -1) + (m -1)] / 3 + 1 

Results of evaluations are stored in experience base in order to learn from the past 
and to use them in new evaluation situations. Global evaluation results of adaptation 
methods, can participate to improve awareness about the limits and strengths of these 
methods. Therefore, this research can contribute towards meta-adaptation support by un¬ 
derstanding the applicability of current methods given user and context characteristics. 
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Conclusion 

Adaptive learning systems are currently lacking evaluation frameworks for measuring 
concretely the quality and effectiveness of adaptation. As layered evaluation ap¬ 
proaches advocate, evaluation practices should take into account components that are 
responsible for adaptive behaviour. Adaptation methods are strongly related to adap¬ 
tive system components. Meta-adaptation, a new research issue, believes that one ad¬ 
aptation method can not be suitable for all contexts. In this paper, we propose a meas¬ 
urement centred evaluation approach focused on adaptation methods assessment. 

The choice of adopted methods in user modelling, presentation and navigation steps 
are a key issue for adaptation performance. Evaluation of the method level is the first 
stage in the assessment of adaptive Web-based learning systems. In our research, we 
propose to explore specific measurement criteria to evaluate adaptation method. 
These criteria are assessed by both learners and experts and they are mainly ex¬ 
pressed by qualitative statements. Global evaluation results are achieved by using 
fuzzy group decision making theory for the aggregation of different decisions. We 
propose to consider current learners and experts judgements related to measure¬ 
ment criteria and also similar past evaluations results in order to reinforce assess¬ 
ment decisions. This work can be used in order to support meta-adaptation. Knowl¬ 
edge base can be constructed, in future works, in order to store different adaptive 
learning situations with related adaptation method feedback. Adaptation process can 
be reinforced and guided to select suitable methods fitting varying contexts. 
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